Simplify installing airflow from GitHub repo in breeze #54070

potiuk · 2025-08-03T17:16:09Z

Breeze had the possibility of installing airflow from a branch of any Github repo - by providing VCS url, but it's been broken since the split of distributions. This PR adds capability of using owner/repo:branch as --use-airflow-version and the installation will be done using this GitHub repo.

Note! Such installation will NOT (currently) compile the assets so you will not be able to run api_server easily after such installation We might want to improve that in the future.

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in airflow-core/newsfragments.

Breeze had the possibility of installing airflow from a branch of any Github repo - by providing VCS url, but it's been broken since the split of distributions. This PR adds capability of using `owner/repo:branch` as `--use-airflow-version` and the installation will be done using this GitHub repo. Note! Such installation will NOT (currently) compile the assets so you will not be able to run api_server easily after such installation We might want to improve that in the future.

gopidesupavan

Cool thanks :)

github-actions · 2025-08-03T20:23:01Z

Backport failed to create: v3-0-test. View the failure log Run details

Status	Branch	Result
❌	v3-0-test

You can attempt to backport this manually by running:

cherry_picker 40ccc55 v3-0-test

This should apply the commit to the v3-0-test branch and leave the commit in conflict state marking
the files that need manual conflict resolution.

After you have resolved the conflicts, you can continue the backport process by running:

cherry_picker --continue

Breeze had the possibility of installing airflow from a branch of any Github repo - by providing VCS url, but it's been broken since the split of distributions. This PR adds capability of using `owner/repo:branch` as `--use-airflow-version` and the installation will be done using this GitHub repo. Note! Such installation will NOT (currently) compile the assets so you will not be able to run api_server easily after such installation We might want to improve that in the future. (cherry picked from commit 40ccc55)

jason810496

Thank, Jarek! Blazing speed, we can now easily set up the unreleased v3-0-test for manual provider testing.

amoghrajesh

Late to the part Jarek, but this is a really nice improvement!

potiuk · 2025-08-05T13:40:39Z

Late to the part Jarek, but this is a really nice improvement!

Cool. I am glad It helps :D

potiuk · 2025-08-05T13:46:21Z

BTW. @amoghrajesh @jason810496 -> I thought about next steps:

add asset compilation after installing airflow (that might be tricky as currently we have no node in the image - so it might requires some tricks. Likely it will require downlading the sources, unpacking them and installing airflow with hatch build -c -t custom after installing node.
adding compat test in CI for v3-0-test - this way at ANY point in time we would know that all "main" providers are tested with "3.0.X in progress" at the tip of our branch.

Might be pretty nice addition to our test harness.

Breeze had the possibility of installing airflow from a branch of any Github repo - by providing VCS url, but it's been broken since the split of distributions. This PR adds capability of using `owner/repo:branch` as `--use-airflow-version` and the installation will be done using this GitHub repo. Note! Such installation will NOT (currently) compile the assets so you will not be able to run api_server easily after such installation We might want to improve that in the future.

jason810496 · 2025-08-10T15:34:17Z

There frontend not built before error when specifing with --use-airflow-version apache/airflow:main( even with start-airflow --dev-mode ).

breeze start-airflow --dev-mode --python 3.10 --backend postgres --integration localstack --mount-sources providers-and-tests --use-airflow-version apache/airflow:main

The Airflow start successfully in Breeze, but the frontend is not built before. So it will result in 500 internal error when accessing the UI due to missing the dist of frontend.

So I still switch to main branch and build distributions then switch back to branch of current PR and run with --use-airflow-version wheel ) as workaround:
#53821 (comment)

I will check where should I add the breeze compile-ui-assets or how to compile the frontend before start-airflow when I have time tomorrow.

potiuk · 2025-08-10T21:08:28Z

I will check where should I add the breeze compile-ui-assets or how to compile the frontend before start-airflow when I have time tomorrow.

Yes- that is a missing thing. The problem with it is that we currently run asset compilation outside of breeze - but the installation with --use-airflow-version already happens fully in breeze. And we have no node/yarn/pnpm installed in CI image.... The problem is that when we install from GitHub URL - we do not have local sources (which would be a different version) so even if you could run breeze command inside breeze (technically such inception shoud be possible- because we also forward docker socket to inside breeze) - what you really need is you need to:

install node + yarn + pnpm
find the directory in site-packages where airflow UI javascript sources are installed
run appropriate npm commands

Not a rocket science but it will take a bit of time to run, and it shoudl be done somewhere here: https://github.com/apache/airflow/blob/main/scripts/in_container/install_airflow_and_providers.py#L767 -> this is where the command to install stuff when --use-airflow-version is actually executed - and "airflow-core" should be installed right after - you should probably naed to add a flag in the InstallationSpec (compile_assets) and set it to true somewhere here - https://github.com/apache/airflow/blob/main/scripts/in_container/install_airflow_and_providers.py#L325 - because this is where we determine what "pip install" to add based on the --use-airflow flag.

Breeze had the possibility of installing airflow from a branch of any Github repo - by providing VCS url, but it's been broken since the split of distributions. This PR adds capability of using `owner/repo:branch` as `--use-airflow-version` and the installation will be done using this GitHub repo. Note! Such installation will NOT (currently) compile the assets so you will not be able to run api_server easily after such installation We might want to improve that in the future.

amoghrajesh · 2025-08-12T05:38:24Z

adding compat test in CI for v3-0-test - this way at ANY point in time we would know that all "main" providers are tested with "3.0.X in progress" at the tip of our branch.

This does make a lot of sense yes.

potiuk · 2025-08-13T11:26:36Z

Converted those two tasks into two issues:

potiuk requested review from amoghrajesh, ashb, gopidesupavan and jedcunningham as code owners August 3, 2025 17:16

boring-cyborg bot added area:dev-tools backport-to-v3-1-test Mark PR with this label to backport to v3-1-test branch labels Aug 3, 2025

potiuk requested review from bugraoz93, jason810496 and rawwar August 3, 2025 17:16

potiuk force-pushed the simplify-installation-from-branch branch from 918c23c to 4de8c73 Compare August 3, 2025 17:24

potiuk force-pushed the simplify-installation-from-branch branch from 4de8c73 to 1badd53 Compare August 3, 2025 17:25

gopidesupavan approved these changes Aug 3, 2025

View reviewed changes

potiuk merged commit 40ccc55 into apache:main Aug 3, 2025
103 checks passed

potiuk deleted the simplify-installation-from-branch branch August 3, 2025 20:22

jason810496 reviewed Aug 4, 2025

View reviewed changes

amoghrajesh reviewed Aug 5, 2025

View reviewed changes

This was referenced Aug 13, 2025

Compile assets when installing airflow from Github Branch in Breeze #54453

Closed

Add compat tests against "v3-X-test" branch tip for providers. #54454

Open

jason810496 mentioned this pull request Oct 25, 2025

Auto-compile UI assets on Breeze start-airflow command #57219

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Simplify installing airflow from GitHub repo in breeze #54070

Simplify installing airflow from GitHub repo in breeze #54070

Uh oh!

potiuk commented Aug 3, 2025

Uh oh!

gopidesupavan left a comment

Uh oh!

Uh oh!

github-actions bot commented Aug 3, 2025

Uh oh!

jason810496 left a comment

Uh oh!

amoghrajesh left a comment

Uh oh!

potiuk commented Aug 5, 2025

Uh oh!

potiuk commented Aug 5, 2025

Uh oh!

jason810496 commented Aug 10, 2025

Uh oh!

potiuk commented Aug 10, 2025

Uh oh!

amoghrajesh commented Aug 12, 2025

Uh oh!

potiuk commented Aug 13, 2025 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Simplify installing airflow from GitHub repo in breeze #54070

Simplify installing airflow from GitHub repo in breeze #54070

Uh oh!

Conversation

potiuk commented Aug 3, 2025

Uh oh!

gopidesupavan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions bot commented Aug 3, 2025

Backport failed to create: v3-0-test. View the failure log Run details

Uh oh!

jason810496 left a comment

Choose a reason for hiding this comment

Uh oh!

amoghrajesh left a comment

Choose a reason for hiding this comment

Uh oh!

potiuk commented Aug 5, 2025

Uh oh!

potiuk commented Aug 5, 2025

Uh oh!

jason810496 commented Aug 10, 2025

Uh oh!

potiuk commented Aug 10, 2025

Uh oh!

amoghrajesh commented Aug 12, 2025

Uh oh!

potiuk commented Aug 13, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

potiuk commented Aug 13, 2025 •

edited

Loading